> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.

# Contrastive Learning

> Understanding how CLIP learns visual-semantic embeddings through contrastive loss

# Contrastive Learning in CLIP

Contrastive learning is the training methodology that enables CLIP to learn aligned visual and semantic representations. The key insight: **maximize agreement between matched image-text pairs while minimizing agreement between mismatched pairs**.

## Core Concept

Given a batch of N (image, text) pairs:

1. Encode all images → N image embeddings
2. Encode all texts → N text embeddings
3. Compute N×N similarity matrix
4. Train to maximize diagonal (correct pairs) and minimize off-diagonal (incorrect pairs)

<Note>
  **Symmetric Loss**: CLIP computes loss from both image→text and text→image directions, ensuring bidirectional alignment.
</Note>

## The Contrastive Loss Function

OpenCLIP implements the contrastive loss in `src/open_clip/loss.py`. The core loss is a **symmetric cross-entropy loss** over the similarity matrix.

### Implementation

From `src/open_clip/loss.py:68-155`:

```python theme={null}
class ClipLoss(nn.Module):
    def forward(
            self,
            image_features,
            text_features,
            logit_scale,
            logit_bias=None,
            output_dict=False,
    ):
        device = image_features.device
        
        # Compute similarity matrix (N×N)
        logits_per_image, logits_per_text = self.get_logits(
            image_features,
            text_features,
            logit_scale,
            logit_bias=logit_bias,
        )

        # Ground truth: diagonal matrix (i-th image matches i-th text)
        labels = self.get_ground_truth(device, logits_per_image.shape[0])

        # Symmetric cross-entropy loss
        total_loss = (
            F.cross_entropy(logits_per_image, labels) +
            F.cross_entropy(logits_per_text, labels)
        ) / 2

        return {"contrastive_loss": total_loss} if output_dict else total_loss
```

### Logits Computation

From `src/open_clip/loss.py:104-130`:

```python theme={null}
def get_logits(self, image_features, text_features, logit_scale, logit_bias=None):
    if self.world_size > 1:
        # Gather features from all GPUs for large batch sizes
        all_image_features, all_text_features = gather_features(
            image_features,
            text_features,
            ...
        )
        logits_per_image = logit_scale * all_image_features @ all_text_features.T
        logits_per_text = logit_scale * all_text_features @ all_image_features.T
    else:
        # Single GPU: compute scaled cosine similarity
        logits_per_image = logit_scale * image_features @ text_features.T
        logits_per_text = logit_scale * text_features @ image_features.T

    if logit_bias is not None:
        logits_per_image += logit_bias
        logits_per_text += logit_bias

    return logits_per_image, logits_per_text
```

## Mathematical Formulation

Given normalized embeddings **I** (images) and **T** (texts):

### Similarity Matrix

```
S = τ · I · T^T
```

Where:

* **τ** (tau) = `logit_scale.exp()` - learnable temperature parameter
* **S\[i,j]** = scaled cosine similarity between i-th image and j-th text

### Loss Function

```
L = 1/2 * [L_i2t + L_t2i]

L_i2t = -1/N * Σ log(exp(S[i,i]) / Σ_j exp(S[i,j]))  # Image to text
L_t2i = -1/N * Σ log(exp(S[i,i]) / Σ_j exp(S[j,i]))  # Text to image
```

This is equivalent to **cross-entropy loss** with ground truth labels on the diagonal.

## Visual-Semantic Embedding Space

Contrastive learning creates a **joint embedding space** where:

### Positive Pairs (Matching)

* Image of "a dog playing fetch" ↔ Text "a dog playing fetch"
* Model learns to embed these **close together**
* High cosine similarity (→ 1.0)

### Negative Pairs (Mismatched)

* Image of "a dog playing fetch" ↔ Text "a cat sleeping"
* Model learns to embed these **far apart**
* Low cosine similarity (→ 0.0 or negative)

### Emergent Properties

Through large-scale contrastive training:

1. **Semantic clustering** - Similar concepts cluster together
2. **Cross-modal alignment** - "dog" (text) aligns with dog images
3. **Compositional understanding** - Model learns objects, actions, attributes
4. **Zero-shot transfer** - Embeddings generalize to unseen concepts

## Training Objective and Batch Construction

### In-Batch Negatives

CLIP uses an efficient strategy: **in-batch negatives**

* Batch size N creates N positive pairs
* Each pair has (N-1) negative examples from other samples
* Total comparisons: N² (N positive + N(N-1) negative)

<Warning>
  **Large batch sizes are critical** for contrastive learning. More negatives = better training signal. OpenCLIP supports batch sizes up to 100K+ across distributed GPUs.
</Warning>

### Batch Construction Example

Given batch size N=4:

```
Images:    [img0, img1, img2, img3]
Texts:     [txt0, txt1, txt2, txt3]

Similarity Matrix (4×4):
        txt0  txt1  txt2  txt3
img0  [HIGH   low   low   low ]  ← img0 matches txt0
img1  [ low  HIGH   low   low ]  ← img1 matches txt1
img2  [ low   low  HIGH   low ]  ← img2 matches txt2
img3  [ low   low   low  HIGH ]  ← img3 matches txt3

Goal: Maximize diagonal, minimize off-diagonal
```

### Ground Truth Labels

From `src/open_clip/loss.py:91-102`:

```python theme={null}
def get_ground_truth(self, device, num_logits) -> torch.Tensor:
    # Ground truth: each image i should match text i
    labels = torch.arange(num_logits, device=device, dtype=torch.long)
    
    if self.world_size > 1 and self.local_loss:
        # Adjust labels for distributed training
        labels = labels + num_logits * self.rank
        
    return labels
```

Labels are simply `[0, 1, 2, ..., N-1]` - each sample matches its corresponding index.

## Advanced Training Techniques

### Local Loss

For distributed training, compute loss locally on each GPU to save memory:

```python theme={null}
if self.local_loss:
    # Only compute gradients for local image features
    logits_per_image = logit_scale * image_features @ all_text_features.T
    logits_per_text = logit_scale * text_features @ all_image_features.T
```

Reduces space complexity from O(n²) to effectively O(n).

### Gather with Gradient

Enable gradient flow during all-gather operation:

```python theme={null}
if gather_with_grad:
    all_image_features = torch.cat(torch.distributed.nn.all_gather(image_features))
    all_text_features = torch.cat(torch.distributed.nn.all_gather(text_features))
```

Allows backpropagation through distributed features.

### SigLIP Loss (Alternative)

OpenCLIP also implements SigLIP loss from `src/open_clip/loss.py:330-464`:

```python theme={null}
class SigLipLoss(nn.Module):
    """ Sigmoid Loss for Language Image Pre-Training (SigLIP) 
    Uses sigmoid instead of softmax for better scaling.
    """
    def _loss(self, image_features, text_features, logit_scale, logit_bias=None):
        logits = self.get_logits(image_features, text_features, logit_scale, logit_bias)
        labels = self.get_ground_truth(...)
        loss = -F.logsigmoid(labels * logits).sum() / image_features.shape[0]
        return loss
```

Benefits:

* Better scaling to very large batches
* No softmax normalization overhead
* Independent per-pair loss computation

## Training Configuration

Example training with contrastive loss:

```bash theme={null}
python -m open_clip_train.main \
    --train-data="/data/laion400m/{00000..41455}.tar" \
    --batch-size=256 \
    --epochs=32 \
    --model=ViT-B-32 \
    --local-loss \        # Enable local loss for memory efficiency
    --gather-with-grad    # Enable gradient gathering
```

### Key Hyperparameters

* **Batch size**: Larger = more negatives = better training (256-32K typical)
* **Learning rate**: 5e-4 to 1e-3 typical for CLIP
* **Warmup**: Gradual learning rate increase (2000-10000 steps)
* **Temperature (τ)**: Learned, initialized to \~2.66

## Loss Curves

During training, monitor:

1. **Contrastive loss** - Should decrease steadily
2. **Accuracy** - Top-1/Top-5 on diagonal predictions
3. **Zero-shot metrics** - Periodic ImageNet zero-shot evaluation

From the README:

> When run on a machine with 8 GPUs the command should produce the following training curve for Conceptual Captions

<Frame>
  <img src="https://raw.githubusercontent.com/mlfoundations/open_clip/main/docs/clip_zeroshot.png" alt="CLIP Zero-Shot Training Curve" />
</Frame>

## Reference Implementation

Key files:

* **`src/open_clip/loss.py`** - ClipLoss, SigLipLoss, CoCaLoss implementations
* **`src/open_clip/model.py:265-480`** - CLIP model with forward pass
* **`src/open_clip_train/train.py`** - Training loop

## Related Concepts

<CardGroup cols={2}>
  <Card title="CLIP Overview" icon="diagram-project" href="/concepts/clip-overview">
    High-level architecture and design principles
  </Card>

  <Card title="Zero-Shot Classification" icon="wand-magic-sparkles" href="/concepts/zero-shot-classification">
    How contrastive embeddings enable zero-shot inference
  </Card>
</CardGroup>

## Further Reading

* **Original CLIP Paper**: [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020)
* **SigLIP Paper**: [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343)
* **OpenCLIP Scaling Laws**: [Reproducible scaling laws for contrastive language-image learning](https://arxiv.org/abs/2212.07143)